Ensemble of deep learning language models to support the creation of living systematic reviews for the COVID-19 literature.

BACKGROUND: The COVID-19 pandemic has led to an unprecedented amount of scientific publications, growing at a pace never seen before. Multiple living systematic reviews have been developed to assist professionals with up-to-date and trustworthy health information, but it is increasingly challenging for systematic reviewers to keep up with the evidence in electronic databases. We aimed to investigate deep learning-based machine learning algorithms to classify COVID-19-related publications to help scale up the epidemiological curation process. METHODS: In this retrospective study, five different pre-trained deep learning-based language models were fine-tuned on a dataset of 6365 publications manually classified into two classes, three subclasses, and 22 sub-subclasses relevant for epidemiological triage purposes. In a k-fold cross-validation setting, each standalone model was assessed on a classification task and compared against an ensemble, which takes the standalone model predictions as input and uses different strategies to infer the optimal article class. A ranking task was also considered, in which the model outputs a ranked list of sub-subclasses associated with the article. RESULTS: The ensemble model significantly outperformed the standalone classifiers, achieving a F1-score of 89.2 at the class level of the classification task. The difference between the standalone and ensemble models increases at the sub-subclass level, where the ensemble reaches a micro F1-score of 70% against 67% for the best-performing standalone model. For the ranking task, the ensemble obtained the highest recall@3, with a performance of 89%. Using an unanimity voting rule, the ensemble can provide predictions with higher confidence on a subset of the data, achieving detection of original papers with a F1-score up to 97% on a subset of 80% of the collection instead of 93% on the whole dataset. CONCLUSION: This study shows the potential of using deep learning language models to perform triage of COVID-19 references efficiently and support epidemiological curation and review. The ensemble consistently and significantly outperforms any standalone model. Fine-tuning the voting strategy thresholds is an interesting alternative to annotate a subset with higher predictive confidence.

Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: Update of a living systematic review and meta-analysis.

BACKGROUND: Debate about the level of asymptomatic Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) infection continues. The amount of evidence is increasing and study designs have changed over time. We updated a living systematic review to address 3 questions: (1) Among people who become infected with SARS-CoV-2, what proportion does not experience symptoms at all during their infection? (2) What is the infectiousness of asymptomatic and presymptomatic, compared with symptomatic, SARS-CoV-2 infection? (3) What proportion of SARS-CoV-2 transmission in a population is accounted for by people who are asymptomatic or presymptomatic? METHODS AND FINDINGS: The protocol was first published on 1 April 2020 and last updated on 18 June 2021. We searched PubMed, Embase, bioRxiv, and medRxiv, aggregated in a database of SARS-CoV-2 literature, most recently on 6 July 2021. Studies of people with PCR-diagnosed SARS-CoV-2, which documented symptom status at the beginning and end of follow-up, or mathematical modelling studies were included. Studies restricted to people already diagnosed, of single individuals or families, or without sufficient follow-up were excluded. One reviewer extracted data and a second verified the extraction, with disagreement resolved by discussion or a third reviewer. Risk of bias in empirical studies was assessed with a bespoke checklist and modelling studies with a published checklist. All data syntheses were done using random effects models. Review question (1): We included 130 studies. Heterogeneity was high so we did not estimate a mean proportion of asymptomatic infections overall (interquartile range (IQR) 14% to 50%, prediction interval 2% to 90%), or in 84 studies based on screening of defined populations (IQR 20% to 65%, prediction interval 4% to 94%). In 46 studies based on contact or outbreak investigations, the summary proportion asymptomatic was 19% (95% confidence interval (CI) 15% to 25%, prediction interval 2% to 70%). (2) The secondary attack rate in contacts of people with asymptomatic infection compared with symptomatic infection was 0.32 (95% CI 0.16 to 0.64, prediction interval 0.11 to 0.95, 8 studies). (3) In 13 modelling studies fit to data, the proportion of all SARS-CoV-2 transmission from presymptomatic individuals was higher than from asymptomatic individuals. Limitations of the evidence include high heterogeneity and high risks of selection and information bias in studies that were not designed to measure persistently asymptomatic infection, and limited information about variants of concern or in people who have been vaccinated. CONCLUSIONS: Based on studies published up to July 2021, most SARS-CoV-2 infections were not persistently asymptomatic, and asymptomatic infections were less infectious than symptomatic infections. Summary estimates from meta-analysis may be misleading when variability between studies is extreme and prediction intervals should be presented. Future studies should determine the asymptomatic proportion of SARS-CoV-2 infections caused by variants of concern and in people with immunity following vaccination or previous infection. Without prospective longitudinal studies with methods that minimise selection and measurement biases, further updates with the study types included in this living systematic review are unlikely to be able to provide a reliable summary estimate of the proportion of asymptomatic infections caused by SARS-CoV-2. REVIEW PROTOCOL: Open Science Framework (https://osf.io/9ewys/).

Outbreaks of publications about emerging infectious diseases: the case of SARS-CoV-2 and Zika virus.

BACKGROUND: Outbreaks of infectious diseases generate outbreaks of scientific evidence. In 2016 epidemics of Zika virus emerged, and in 2020, a novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) caused a pandemic of coronavirus disease 2019 (COVID-19). We compared patterns of scientific publications for the two infections to analyse the evolution of the evidence. METHODS: We annotated publications on Zika virus and SARS-CoV-2 that we collected using living evidence databases according to study design. We used descriptive statistics to categorise and compare study designs over time. RESULTS: We found 2286 publications about Zika virus in 2016 and 21,990 about SARS-CoV-2 up to 24 May 2020, of which we analysed a random sample of 5294 (24%). For both infections, there were more epidemiological than laboratory science studies. Amongst epidemiological studies for both infections, case reports, case series and cross-sectional studies emerged first, cohort and case-control studies were published later. Trials were the last to emerge. The number of preprints was much higher for SARS-CoV-2 than for Zika virus. CONCLUSIONS: Similarities in the overall pattern of publications might be generalizable, whereas differences are compatible with differences in the characteristics of a disease. Understanding how evidence accumulates during disease outbreaks helps us understand which types of public health questions we can answer and when.

Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: A living systematic review and meta-analysis.

BACKGROUND: There is disagreement about the level of asymptomatic severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. We conducted a living systematic review and meta-analysis to address three questions: (1) Amongst people who become infected with SARS-CoV-2, what proportion does not experience symptoms at all during their infection? (2) Amongst people with SARS-CoV-2 infection who are asymptomatic when diagnosed, what proportion will develop symptoms later? (3) What proportion of SARS-CoV-2 transmission is accounted for by people who are either asymptomatic throughout infection or presymptomatic? METHODS AND FINDINGS: We searched PubMed, Embase, bioRxiv, and medRxiv using a database of SARS-CoV-2 literature that is updated daily, on 25 March 2020, 20 April 2020, and 10 June 2020. Studies of people with SARS-CoV-2 diagnosed by reverse transcriptase PCR (RT-PCR) that documented follow-up and symptom status at the beginning and end of follow-up or modelling studies were included. One reviewer extracted data and a second verified the extraction, with disagreement resolved by discussion or a third reviewer. Risk of bias in empirical studies was assessed with an adapted checklist for case series, and the relevance and credibility of modelling studies were assessed using a published checklist. We included a total of 94 studies. The overall estimate of the proportion of people who become infected with SARS-CoV-2 and remain asymptomatic throughout infection was 20% (95% confidence interval [CI] 17-25) with a prediction interval of 3%-67% in 79 studies that addressed this review question. There was some evidence that biases in the selection of participants influence the estimate. In seven studies of defined populations screened for SARS-CoV-2 and then followed, 31% (95% CI 26%-37%, prediction interval 24%-38%) remained asymptomatic. The proportion of people that is presymptomatic could not be summarised, owing to heterogeneity. The secondary attack rate was lower in contacts of people with asymptomatic infection than those with symptomatic infection (relative risk 0.35, 95% CI 0.10-1.27). Modelling studies fit to data found a higher proportion of all SARS-CoV-2 infections resulting from transmission from presymptomatic individuals than from asymptomatic individuals. Limitations of the review include that most included studies were not designed to estimate the proportion of asymptomatic SARS-CoV-2 infections and were at risk of selection biases; we did not consider the possible impact of false negative RT-PCR results, which would underestimate the proportion of asymptomatic infections; and the database does not include all sources. CONCLUSIONS: The findings of this living systematic review suggest that most people who become infected with SARS-CoV-2 will not remain asymptomatic throughout the course of the infection. The contribution of presymptomatic and asymptomatic infections to overall SARS-CoV-2 transmission means that combination prevention measures, with enhanced hand hygiene, masks, testing tracing, and isolation strategies and social distancing, will continue to be needed.